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Validation  Report  Lanes  Training 


INTRODUCTION 


Background 

The  Advanced  Research  Projects  Agency  (ARP A)  and  the  Army  National  Guard 
(ARNG)  contracted  with  BDM  Federal,  Inc.  to  develop  a  series  of  computer-based  eourses  to 
instruct  soldiers  in  the  supply,  maintenanee  and  medieal  companies  of  the  Forward  Support 
Battalion  (FSB).  ARNG  subject-matter  experts  developed  the  concept  of  FSB  Training  Lanes 
which  they  used  to  identify  the  critieal  skills  that  could  be  taught  best  by  applieation  of 
computer-based  instruction  (CBI).  The  use  of  the  lanes  training  concept  to  identify  the  course 
content  is  described  in  a  subsequent  section  of  this  report.  The  development  of  CBI  for  this 
content  is  described  in  detail  by  Deterline  (1995). 

All  CBI  eourses  developed  for  the  FSB  were  subject  to  a  formative  evaluation. 

Bloom,  Hastings  and  Madaus  {Handbook  on  Formative  and  Summative  Evaluation  of  Student 
Learning,  New  York:  McGraw-Hill,  1971)  describe  formative  evaluation  of  a  eurriculum  in 
this  way: 


.  .  .  formative  evaluation  involves  the  collection  of  appropriate 
evidence  during  the  construction  and  trying  out  of  a  new 
eurriculum  in  such  a  way  that  revisions  of  the  eurriculum  can  be 
based  on  the  evidence,  (p.  117) 

In  the  development  of  the  FSB  courses,  formative  evaluation  began  in  the  design 
phase.  Subject-matter  experts  and  instruetional  designers  examined  the  content  and 
sequencing  to  determine  if  they  were  appropriate  on  a-priori  grounds.  This  internal  quality 
control  process  was  followed  by  a  phase  of  ‘Alpha  testing’  in  which  the  developers  performed 
a  trial-run  of  the  lesson  to  determine  if  the  courseware  tracked  with  the  storyboards,  and  to 
identify  flaws  in  the  logic  of  presentation  and  errors  or  ambiguities  in  the  instructions  for 
navigation  through  the  eourse.  Finally,  a  phase  of  ‘Beta  testing’  used  real  subjects  drawn 
from  the  pool  of  target  users  to  determine  if  they  eould  navigate  through  the  eourse 
suceessfully  and  the  extent  to  which  they  learned  from  the  material. 

Purpose 

This  report  documents  the  selection  of  course  content  and  the  Beta  testing  performed 
on  the  CBI  modules  developed  to  instruct  personnel  in  the  FSB  companies.  The  report 
documents  the  performance  gains  (on  computer-based  tests)  attained  by  the  test  subjeets,  and 
discusses  some  of  the  eharacteristics  of  the  lessons  that  appear  to  be  related  to  their  relative 
effectiveness. 
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IDENTIFICATION  OF  TRAINING  CONTENT 


Lanes  Training  Concept 

The  Iowa  National  Guard  has  developed  the  lanes  training  concept  for  application  to 
the  supply,  maintenance,  and  medical  companies  of  the  FSB.  The  lanes  training  concept  is 
borrowed  from  the  idea  of  the  training  lane  used  by  the  combat  arms.  A  training  lane  for  a 
combat  arms  unit  involves  a  physical  lane  on  the  ground  through  which  the  combat  vehicles 
pass  while  being  subjected  to  various  threats  that  they  must  react  to  and  defeat.  The  lanes 
training  concept  applied  to  an  FSB  company  consists  of  identifying  critical  collective  tasks 
performed  during  an  interval  of  time  and  arranging  these  tasks  into  a  Tane’  consisting  of  a 
series  of  ‘training  stations’  that  takes  the  unit  from  the  beginning  to  the  end.  The  critical 
collective  tasks  are  identified  by  examining  the  Mission  Essential  Task  List  (METL)  of  a 
particular  FSB  company. 

First  applied  to  the  Maintenance  Company,  the  Iowa  National  Guard  laid  out  training 
stations  supporting  a  training  lane  in  such  a  way  that  the  equipment,  personnel,  and  time 
requirements  for  each  station  were  fully  specified.  Since  the  entire  training  lane  would 
normally  require  more  than  one  weekend  of  training  time,  modularization  of  the  training  lane 
into  training  stations  allowed  the  persoimel  being  trained  to  work  on  the  training  stations  over 
a  sequence  of  weekends.  This  notion  of  modularization  was  carried  forward  into  the  CBI 
versions  of  the  training  lanes. 

Identification  of  Training  Lanes 

ARPA  requested  that  the  Iowa  National  Guard  provide  subject  matter  experts  (SMEs) 
to  assist  in  the  identification  of  appropriate  training  lanes  for  the  supply,  maintenance,  and 
medical  companies  of  the  FSB.  These  SMEs  performed  a  front-end  analysis  that  resulted  in 
identifying  two  potential  training  lanes  for  each  of  these  companies.  In  addition,  data  from 
the  National  Training  Center  (NTC)  indicated  that  the  FSB  companies  were  all  experiencing 
difficulties  in  establishing  defensive  positions.  Consequently,  a  common  Defend  Company 
Area  training  lane  was  developed.  Ultimately,  one  of  the  possible  lanes  specific  to  each 
company  was  selected  for  further  development.  Table  1  indicates  which  collective  task  was 
chosen  for  each  company. 


Table  1:  Company-Specific  Collective  Tasks  Selected  for  Development 


Company 

METL  Task 

Selected  Supporting  Collective  Task 
(Training  Lane) 

Supply 

Request,  Receive,  and  Issue  Supplies 

Provide  Class  III  (Bulk)  Supplies 

Maintenance 

Conduct  Direct  Support  Maintenance  and 
Repair  Parts  Supply  Service  Operations 

Provide  On  Site  Maintenance 

Medical 

Perform  Health  Service  Support  Operations 

Provide  Ground  Ambulance 
Evacuation  Support 
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Table  2:  Training  Lanes  and  Lessons  for  CBI  Development 


Training  Lane  Number  of 

Lesson  Name  Beta  Tests 

PROVIDE  CLASS  III  (BULK)  SUPPLIES 

A1  Supervise  Receipt/Storage  of  POL  Products  3 

A2  Inspection  of  POL  Products  2 

A3  Direct  POL  Environment/Security  Controls  2 

A4  Tanker  Operation  and  Safety  2 

A5  Tank  Trailer  Operations  1 

PROVIDE  ON  SITE  MAINTENANCE 

B1  Inspect/Troubleshoot  Track  Vehicle  Auto  Systems  1 

B2  Repair  Diesel  Power  Pack  5 

B3  Troubleshoot/Repair  Radio  Sets  3 

B4  Repair  Traversing  Systems  3 

B5  Repair  BFV  TOW2  ISU  3 

B6  Organize  and  Dispatch  Maintenance  Support  Team  3 

B7  Supervise/Perform  BDAR  2 

PROVIDE  GROUND  AMBULANCE  EVACUATION  SUPPORT 

C1  Control  Bleeding  4 

C2  Triage/Survey  Patients  4 

C3  Plan  for  Evacuation  2 

C4  Treating  Respiratory  Dysfunction  1 

C5  General  Casualty  Management  3 

C6  Treatment  of  Wounds  3 

DEFEND  COMPANY  AREA 

D1  Terrain  Analysis  2 

D2  Commander:  Plan  Sector  Defense  6 

D3  Commander:  Prepare  Support  Plans  4 

D4  Commander:  Prepare  for  Engagement  3 

D5  Company:  Prepare  for  Engagement  2 

D6  Commander:  Organize  Hasty  Defense/Disengagement  3 

D7  Company;  Conduct  Hasty  Defense/Disengagement  3 
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ARP  A  directed  the  SMEs  supporting  this  project  to  identify  specific  skills/tasks  within 
each  lane  that  would  be  useful  to  have  in  CBI  format.  The  guidance  was  to  choose  tasks  that 
are  difficult  to  teach  in  the  Armory  and  that  multimedia  CBI  should  be  able  to  instruct.  The 
goal  was  to  supply  multimedia  instruction  that  would  be  an  effective  substitute  for  initial 
levels  of  hands-on  instruction.  Table  2  displays  the  final  set  of  training  lanes  and  CBI 
lessons  for  this  project. 

The  CBI  lessons  are  components  of  each  training  lane.  To  conduct  the  entire  lane  the 
training  manager  must  employ  text-based,  CBI,  and  hands-on  instruction.  Additional 
materials  providing  guidance  to  training  managers  about  implementing  the  entire  lane  have 
been  developed  (see  references  under  ARP  A/Iowa  National  Guard).  The  CBI  lessons  are  a 
critical  component  of  these  lanes.  For  the  lane  to  function  properly,  the  CBI  must  provide  the 
students  an  opportunity  to  acquire  the  skills  well  enough  that  he  or  she  will  be  prepared  to 
master  them  in  the  limited  number  of  hands-on  experiences  that  can  be  provided.  Without 
extensive  testing,  it  is  difficult  to  establish  the  exact  degree  of  mastery  required  in  the  CBI 
environment  to  facilitate  transfer  of  training  to  the  real  world.  One  purpose  of  the  Beta 
testing  conducted  as  part  of  the  formative  evaluation  of  the  CBI  lessons  was  to  establish  the 
degree  to  which  students  appeared  to  learn  from  the  CBI.  As  further  experience  is  gathered 
on  the  performance  of  students  in  the  subsequent  hands-on  sections  of  the  complete  lanes, 
then  some  inferences  can  be  made  about  how  well  the  CBI  training  transfers  to  the  real 
world. 


ANALYSIS  OF  BETA  TEST  DATA 


Data 


Appendix  A  lists  the  data  available  for  this  validation  study.  Originally,  it  was  hoped 
that  there  would  be  at  least  three  subjects  tested  for  each  of  the  lessons.  Unfortunately,  there 
were  not  sufficient  numbers  of  personnel  in  appropriate  MOS  designations  available  in  the 
geographic  region  where  the  CBI  was  developed.  Of  the  25  lessons,  15  (60%)  had  three  or 
more  test  subjects,  while  7  other  lessons  had  two  test  subjects,  and  3  had  only  one  test 
subject. 

A  total  of  74  beta  tests  were  performed  in  this  validation.  For  each  beta  test,  the 
student  went  through  a  pretest  --  lesson  -  posttest  sequence.  Typically,  they  went  through 
this  sequence  in  one  sitting  (sometimes  interrupted  for  a  meal  break). 

The  instruction  is  designed  to  increase  the  student’s  knowledge  of  the  subject  matter, 
and  to  increase  the  number  of  subjects  who  are  able  to  master  the  content  of  the  lessons.  To 
examine  the  degree  to  which  the  lessons  were  effective,  we  analyzed  two  types  of 
performance  measures:  the  percentage  correct  at  pretest  and  posttest,  and  the  attainment  of 
mastery  at  pretest  and  posttest.  For  the  purpose  of  this  study,  mastery  is  defined  as  attaining 
a  score  of  80  percent  or  higher  on  the  proficiency  test. 

The  CBI  allows  for  the  possibility  that  a  person  taking  the  lesson  will  have  already 
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mastered  some  components  of  the  instruction  (called  topics).  Students  who  show  mastery  of 
topics  at  the  time  of  the  pretest  are  not  required  to  take  either  the  instruction  or  the  posttest 
on  that  topic.  However,  during  the  beta  testing  some  of  the  subjects  were  encouraged  to  take 
the  instruction  and  all  items  on  the  posttest  so  that  the  development  team  could  be  sure  that 
the  navigation  through  these  sections  was  correct  and  unambiguous.  In  scoring  these  results, 
a  rule  was  imposed  that  if  a  person  was  allowed  to  skip  the  posttest  in  a  particular  topic,  he 
or  she  was  credited  with  the  pretest  score  on  those  topics  (usually  100%).  Thus,  the  posttest 
scores  and  percentages  reported  here  are  based  on  the  assumption  that  a  student  who  knew 
the  material  at  the  beginning  of  the  lesson  will  not  have  forgotten  the  material  (or  have  been 
confused  by  the  instruction)  by  the  end  of  the  lesson. 

Interviews  were  conducted  after  the  testing.  In  addition  to  eliciting  general  comments, 
these  interviews  focused  on  two  important  characteristics  of  the  lessons:  the  degree  to  which 
the  subjects  understood  the  instructions  about  what  to  do  next,  and  the  appropriateness  of  the 
level  of  vocabulary.  Another  important  characteristic  of  the  lessons,  the  time  it  takes  to 
complete  the  material  (including  testing  before  and  after  the  instruction),  was  recorded  as  the 
subjects  went  through  the  lesson.  In  the  following  analyses,  these  characteristics  of  the 
lessons  are  related  to  the  performance  of  the  students. 

Analysis 


Some  of  the  students  who  participated  in  the  Beta  testing  were  already  relatively 
expert  in  the  content  of  the  lessons.  They  were  chosen  to  perform  a  particularly  careful 
review  of  the  content.  Altogether,  there  were  8  instances  in  which  a  student  was  able  to 
attain  the  nominal  mastery  level  (80  percent  or  higher  correct)  on  the  pretest:  two  students 
attained  this  level  on  one  of  the  Maintenance  lessons,  and  six  other  instances  occurred  on 
lessons  in  the  Defend  Company  Sector  lane.  No  student  who  showed  mastery  at  the  time  of 
the  pretest  declined  to  a  level  below  mastery  at  the  time  of  the  posttest.  The  two  pretest 
masters  in  Maintenance  had  a  pretest  average  of  86.5  percent  correct  and  a  posttest  average  of 
89.5  on  the  lesson  they  had  mastered.  The  six  tests  with  mastery  scores  on  lessons  in  Defend 
Company  Sector  averaged  83.7  percent  correct  on  pretest  and  96.7  percent  correct  on  the 
posttest.  The  following  discussion  concerns  only  those  beta  tests  where  the  students  did  not 
attain  mastery  at  the  pretest’. 

Table  3  illustrates  two  ways  of  examining  the  degree  to  which  the  lessons  within  a 
lane  were  effective  in  instructing  the  test  students:  increasing  knowledge  of  subject  matter  and 
percentage  attaining  mastery.  Students  who  were  not  masters  at  the  time  of  the  pretest 
learned  a  considerable  amount  from  the  lessons.  Scores  of  non-masters  averaged  about  58 
percent  correct  before  instruction  and  about  84  percent  correct  after  instruction.  Students  in 
all  lanes  made  considerable  gains  from  taking  the  lessons. 

Table  3  also  shows  that  for  the  Supply,  Medical  and  Defend  Company  Sector  lanes 
large  percentages  (at  least  76  percent)  of  the  students  who  were  not  masters  at  the  pretest  had 


The  posttest  scores  were  not  recorded  for  four  students,  so  results  are  based  on  62  tests. 
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attained  that  level  by  the  time  they  completed  the  lesson  and  took  the  posttest.  The 
Maintenance  lessons  are  notably  different  in  this  regard,  with  only  33  percent  of  the  non¬ 
masters  moving  to  mastery. 

Table  3:  Average  Pretest  and  Posttest  Scores  (for  non-masters  at  pretest),  by  Training  Lane 


Training  Lane 

Average  Pretest 
Score 

Average  Posttest 
Score 

Percent  of 
Masters  at 
Posttest 

Number  of 
Beta  Tests 

Supply 

54.1 

82.7 

80.0 

10 

Maintenance 

50.5 

75.4 

33.3 

18 

Medical 

63.2 

87.5 

76.5 

17 

Defend  Company  Sector 

62.4 

90.0 

88.2 

17 

Total 

57.8 

83.9 

67.7 

62 

Table  4  shows  that  a  very  high  percentage  of  the  students  taking  these  lessons  reported 
that  the  level  of  vocabulary  was  appropriate.  However,  the  lessons  differed  in  the  clarity  of 
the  directions  in  the  CBI.  The  Medical  lane  CBI  had  the  lowest  proportion  of  students 
reporting  that  the  instructions  were  clear.  Generally,  however,  this  lack  of  clarity  did  not 
seem  to  affect  the  attainment  of  the  students.  Analysis  of  their  comments  indicated  that  they 
were  reporting  that  some  sequences  required  considerably  more  attention  than  others  to 
determine  what  they  were  supposed  to  do  next.  This  might  be  a  characteristic  to  reexamine  if 
it  becomes  necessary  to  rewrite  these  lessons  because  of  changes  to  doctrine  or  equipment. 
Clarity  of  the  instructions  was  above  average  for  the  lessons  in  the  Maintenance  lane,  so  this 
does  not  account  for  the  difference  in  attainment  of  mastery. 


Table  4:  Appropriateness  of  Vocabulary  and  Clarity  of  Instructions,  by  Training  Lane 


Training  Lane 

Percent  Reporting 
Vocabulary  was 
Appropriate 

Percent  Reporting 
Instructions  were  Clear 

Number  of  Beta 
Tests 

Supply 

100.0 

60.0 

10 

Maintenance 

94.1 

64.7 

17 

Medical 

87.5 

43.8 

16 

Defend  Company  Sector 

100.0 

75.0 

16 

Total 

94.9 

61.0 

59 

Table  5  shows  that  the  lanes  differed  considerably  in  the  length  of  time  required  to  go 
through  the  lessons.  Lessons  in  the  Maintenance  lane  averaged  nearly  1.5  hours  in  length 
(almost  2.0  hours  including  testing),  while  those  in  the  other  lanes  averaged  less  than  one 
hour  (about  1.25  hours,  with  testing).  Table  5  also  indicates  that  the  students  taking  the 
Maintenance  lane  knew  less  of  the  material  initially.  These  characteristics  may  be  important 
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in  determining  the  rate  at  which  the  Maintenance  lane  lessons  produced  mastery  by  the  time 
of  the  posttest. 


Table  5:  Minutes  Required  to  Take  One  Lesson,  by  Training  Lane 


Training  Lane 

Minutes  to  complete 
lesson  and  testing 

Minutes  to  complete 
lesson  alone 

Number  of 
Beta  Tests 

Supply 

73.8  (25.3) 

51.4  (18.9) 

10 

Maintenance 

119.2  (47.6) 

89.3  (38.4) 

18 

Medical 

79.5  (29.9) 

53.2  (22.2) 

17 

Defend  Company  Sector 

75.8  (40.8) 

45.8  (21.7) 

16 

Total 

89.3  (42.2) 

61.6  (32.5) 

61 

Table  note:  Numbers  in  parentheses  are  standard  deviations. 


Table  6  examines  the  individual  lessons  within  the  Maintenance  lane  in  more  detail. 
It  shows  the  length  time  required  to  take  the  lessons,  the  average  pretest  and  posttest  scores 
on  the  lanes,  and  the  proportion  of  non-masters  who  attained  mastery.  This  table  indicates 
that  three  of  the  lessons  in  this  lane  (B3,  B5,  B6)  were  somewhat  like  the  typical  lessons  in 
the  other  lanes:  average  pretest  scores  were  above  60  percent  correct,  and  the  lessons  took 
1.0- 1.5  hours  to  complete  (including  testing).  Seventy-one  percent  of  the  students  who  were 
initially  non-masters  attained  mastery  after  taking  these  lessons.  This  is  a  somewhat  lower 
rate  than  that  for  the  other  lanes. 


Table  6:  Detail  on  Lessons  in  the  Maintenance  Training  Lane 


Lesson 


B1 


B2 


B3 


B4 


B5 


B6 


B7 


Minutes  to 
Complete  Lesson 
and  Testing 


180.0  (  0.0) 


158.4  (15.0) 


91.0  (24.3) 


156.7  (10.4) 


78.0  (  0.0) 


78.0  (45.9) 


59.0  (  8.5) 


Table  note:  Numbers  in  parentheses  are  standard 


Minutes  to 
Complete  Lesson 
Alone 


140.0  (  0.0) 


115.6  (11.4) 


71.7  (20.8) 


123.3  (11.5) 


62.0  (  0.0) 


56.0  (42.2) 


37.5  ( 4.9) 


Average 

Pretest 

Score 


47.0 


40.0 


68.3 


31.0 


74.0 


68.3 


42.5 

deviations. 


Average 

Posttest 

Score 


67.0 


67.4 


86.0 


68.7 


84.0 


83.3 


77.5 


Percent  of 
Masters 
(Posttest) 


0.0 


0.0 


66.7 


0.0 


100.0 


66.7 


50.0 


Number  of 
Beta  Tests 


1 


One  lesson  (B7)  had  low  pretest  scores,  but  could  be  completed  in  a  relatively  short 
time.  Half  of  the  students  who  had  not  previously  mastered  this  lesson  were  able  to  master  it 
by  taking  the  lesson.  Students  taking  the  other  three  lessons  (Bl,  B2,  B4)  had  very  low 
average  pretest  scores  and  required  very  long  times  to  complete  the  lessons  (about  2  hours). 
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Perhaps  the  topies  covered  in  these  lessons  are  too  unfamiliar  to  the  students  and  the  time 
required  to  assimilate  this  knowledge  is  simply  too  long  to  allow  the  students  to  move  from 
non-mastery  to  mastery  in  a  single  session. 

DISCUSSION 

Generally  speaking,  the  CBl  lessons  developed  to  train  soldiers  in  the  supply, 
maintenance  and  medical  companies  of  the  FSB  are  highly  effective.  Beta  test  results  show 
that  posttest  scores  are  considerably  higher  than  pretest  scores  and  two-thirds  of  the  students 
who  were  non-masters  at  the  time  of  the  pretest  became  masters  after  taking  the  lesson. 

Three  lessons  in  the  Maintenance  lane  were  different  from  the  others  in  that  students  began 
with  very  little  knowledge  of  these  topics,  the  lessons  took  very  much  longer  to  complete,  and 
no  students  were  able  to  master  the  material  in  these  lessons.  It  may  be  that  the  length  of  the 
lessons  inhibited  the  students  from  attaining  mastery  when  they  attempted  to  take  the  lesson 
in  a  single,  extended  session.  Because  the  CBI  allows  students  to  take  the  lessons  topic-by- 
topic,  students  should  be  able  to  acquire  mastery  by  pacing  themselves  through  the  lessons, 
rather  than  taking  an  entire  long  lesson  at  one  sitting. 

Table  7  is  provided  as  additional  guidance  for  training  managers  and  students.  It 
shows  the  average  time  required  by  the  beta  test  students  to  take  each  lesson  (including 
testing)  and  the  average  posttest  score^.  These  figures  can  be  used  during  instructional 
planning  to  determine  whether  a  student  will  have  the  time  to  take  a  lesson,  and  to  make  a 
judgement  about  the  proficiency  to  be  gained  by  taking  the  lesson.  Appendix  C  provides 
additional  information  on  student  attainment. 

RECOMMENDATIONS 

The  CBI  permits  these  lessons  to  be  used  in  a  modular  fashion,  and  it  may  be  wise  to 
advise  training  managers  and  the  students  taking  the  lessons  that  if  they  are  xmfamiliar  with 
the  material  at  the  start  (evidenced  by  a  pre-test  score  below  50  percent  correct),  or  if  the 
lesson  is  very  long  (more  than  80  minutes),  they  should  plan  to  take  it  in  two  or  three 
sessions  rather  than  attempt  to  complete  it  all  at  one  sitting. 


Appendix  B  indicates  that  the  SMEs’  original  estimates  of  time  required  for  these  lessons 
is  not  accurate,  so  the  actual  times  derived  from  the  beta  testing  are  provided  for  more 
accurate  guidance  about  time  required  to  complete  each  lesson. 


8 


Table  7:  Time  Required  and  Posttest  Scores  for  CBI  Lessons,  by  Training  Lane 


Training  Lane 

Total 

Posttest 

Lesson  Name 

Minutes 

Percent 

PROViDE  CLASS  III  (BULK)  SUPPLIES 

Required 

Correct 

A1  Supervise  Receipt/Storage  of  POL  Products 

83 

87 

A2  Inspection  of  POL  Products 

78 

75 

A3  Direct  POL  Environment/Security  Controls 

53 

85 

A4  Tanker  Operation  and  Safety 

62 

82 

A5  Tank  Traiier  Operations 

104 

83 

PROVIDE  ON  SITE  MAINTENANCE 

B1  Inspect/Troubleshoot  Track  Vehicle  Auto  Systems 

180 

67 

B2  Repair  Diesel  Power  Pack 

158 

67 

B3  Troubleshoot/Repair  Radio  Sets 

91 

86 

B4  Repair  Traversing  Systems 

157 

69 

B5  Repair  BFV  TOW2  ISU 

78 

84 

B6  Organize  and  Dispatch  Maintenance  Support  Team 

78 

83 

B7  Supervise/Perform  BDAR 

59 

77 

PROVIDE  GROUND  AMBULANCE  EVACUATION  SUPPORT 

C1  Control  Bleeding 

46 

94 

C2  Triage/Survey  Patients 

103 

82 

C3  Plan  for  Evacuation 

60 

100 

C4  Treating  Respiratory  Dysfunction 

93 

72 

C5  General  Casualty  Management 

103 

88 

C6  Treatment  of  Wounds 

79 

83 

DEFEND  COMPANY  AREA 

D1  Terrain  Analysis 

70 

90 

D2  Commander:  Plan  Sector  Defense 

110 

83 

D3  Commander:  Prepare  Support  Plans 

92 

88 

D4  Commander:  Prepare  for  Engagement 

58 

88 

D5  Company:  Prepare  for  Engagement 

44 

100 

D6  Commander:  Organize  Hasty  Defense/Disengagement 

39 

100 

D7  Company:  Conduct  Hasty  Defense/Disengagement 

40 

95 
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APPENDIX  A 
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ANALYSIS  OF  CBI  BETA  TEST  RESXILTS 
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APPENDIX  R 


Estimation  of  Time  Required  to  Complete  Lessons 

To  make  projections  about  the  time  that  would  be  required  to  develop  the  CBI 
instruction,  the  SMEs  estimated  the  amount  of  time  that  would  be  required  for  a  student  to 
take  each  lesson.  These  estimates  were  made  prior  to  starting  on  the  development  of 
storyboards,  when  only  the  basic  content  of  each  lesson  had  been  decided  upon.  The  SMEs 
were  not  experienced  in  CBI  development  so  they  based  their  estimates  on  their  experience 
with  other  forms  of  instruction. 


Because  the  courses  were  developed  to  be  used  as  modules,  it  is  difficult  to  predict  the 
time  required  for  an  individual  student  to  take  a  lesson.  A  student  may  find,  fi-om  the  pretest, 
that  he  or  she  knows  a  lot  about  certain  topics  covered  in  a  lesson  and  will  be  allowed  to  skip 
this  instruction  and  the  associated  posttest.  Other  students  will  take  all  of  the  topics.  When 
the  SMEs  developed  their  estimates,  they  assumed  that  students  would  take  all  of  the 
modules.  They  did  not  explicitly  include  time  to  take  pretests  and  posttests  (which  are 
integral  components  of  the  CBI  instruction)  in  their  estimates. 


As  development  progressed,  some  lessons  were  shorter  than  anticipated,  while  others 
were  longer.  Some  content  was  reorganized  to  facilitate  instruction.  The  original  lesson  for 
Tanker  Operation  and  Safety  was  split  into  a  lesson  of  the  same  name  and  one  on  Tank 
Trailer  Operations.  The  original  lesson  on  Defend/Prepare  for  Level  I,  II,  and  III 
Threat/ Attack  was  split  into  a  lesson  addressing  the  company  commander  and  one  addressing 
other  company  personnel.  The  original  lessons  Perform  Withdrawal  Under  Fire  and  Conduct 
Hasty  Displacement  contained  a  mixture  of  tasks  for  leaders  and  other  personnel  and  these 
were  sorted  out  into  two  separate  lessons.  In  Table  B-1  the  time  (in  minutes)  projected  for 
the  aggregate  of  the  lessons  in  each  lane  is  contrasted  with  the  actual  total  time  required  to 
complete  the  lessons  and  testing.  Since  some  of  the  beta  test  students  did  not  have  to  take  all 
the  topics  and  posttests  (because  they  obtained  high  scores  on  topic  pretests),  the  actual  times 
underestimate  the  total  amount  of  CBI  developed  in  this  project.  Taking  all  lanes  together, 
the  actual  time  exceeds  the  estimate  by  30  percent. 


Table  B-1;  Estimated  and  Actual  Time  Required  to  Take  CBI  Courses  and  Tests 


Training  Lane 


Estimated  Time® 


Actual  Time’’ 


Supply 

345 

380 

Maintenance 

570 

800 

Medical 

480'= 

484“ 

Defend  Company  Area 

240“ 

453“ 

Actual  time  (in  minutes)  is  the  sum  of  the  average  time  required  by  beta  test  students  on  each  lesson 
in  the  lane. 

Medical  and  Defend  Company  Area  times  do  not  include  the  collective  lesson  for  each  lane. 
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The  next  set  of  figures  examines  the  accuracy  of  the  SME  estimates  by  relating  the 
estimated  time  to  the  average  actual  time  to  complete  each  lesson.  For  the  lessons  that  were 
split,  the  estimated  time  for  the  original  lesson  is  related  to  the  sum  of  the  actual  times  for  the 
two  lessons.  The  estimated  times  for  the  two  defend  lessons  that  were  reorganized  were 
equal  and  it  was  assumed  that  the  reorganization  resulted  in  two  lessons  of  equal  length. 

Figures  B-1  shows  the  relationship  between  the  estimated  and  the  actual  times  to 
complete  the  lesson  alone,  while  figure  B-2  shows  the  relationship  between  the  estimated  and 
actual  times  to  complete  both  the  lesson  and  the  testing.  All  times  are  given  in  minutes.  The 
actual  times  are  the  average  times  required  by  the  students  who  beta  tested  these  lessons. 

The  regression  line  for  the  relationship  in  each  plot  is  defined  by  the  endpoints  marked  with 
the  letter  ‘R’.  Both  plots  are  characterized  by  having  four  very  strong  outliers  —  values  for 
which  the  actual  time  is  much  higher  than  the  estimated  time.  These  points  tend  to  diminish 
the  relationship  between  estimated  and  actual  times,  neither  of  which  is  statistically 
significant.  This  means  that  based  on  the  entire  set  of  data,  the  SMEs  predictions  about  time 
requirements  were  not  related  to  the  actual  times. 

Figures  B-3  and  B-4  show  that  if  the  four  strong  outliers  are  removed,  there  is  a 
substantial  positive  relationship  between  the  SMEs’  estimated  times  and  the  actual  times 
required  to  complete  these  lessons  (not  including  testing).  The  estimated  time  correlates  .53 
with  the  lesson  time,  but  only  .26  with  the  total  time.  The  first  of  these  correlations  is 
statistically  significant.  These  relationships  indicate  that  (with  a  few  exceptions  -  the 
outliers)  the  SMEs  were  able  to  predict  which  lessons  would  take  more  time  than  others. 

They  were,  however,  less  able  precisely  to  predict  the  total  amount  of  time  required  to  take 
the  tests  and  the  lessons.  Thus,  they  underestimated  the  amount  of  CBI  development  that 
would  be  required  for  this  project. 

Because  each  student  will  come  to  a  lesson  with  different  preexisting  knowledge  of 
the  subject  matter,  and  because  the  lessons  allow  students  who  exhibit  mastery  at  the  time  of 
the  pretest  to  skip  certain  instruction  and  posttesting,  it  is  very  difficult  to  predict  how  much 
time  an  individual  student  will  take.  We  recommend  that  training  managers  use  the  average 
actual  times  (see  Table  7  in  the  main  section  of  this  document)  as  guidance,  and  be  prepared 
with  a  contingency  activity  for  students  who  finish  more  rapidly  than  these  times  predict. 

The  modular  nature  of  the  instruction  will  allow  a  student  who  requires  more  time  to 
complete  some  topics  at  one  session  and  finish  the  rest  at  another  session. 
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FIGURE  B-1:  ACTUAL  AND  ESTIMATED  TIME  TO  TAKE  LESSON  (TESTING  NOT  INCLUDED) 
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23  cases  plotted.  Regression  statistics: 

Correlation  .14173  R  Squared  .02009  S.E.  of  Est  31.12899  Sig.  .5189 
Intercept (S .E. )  53,40442(  16.36494)  Slope(S.E.)  .13865(  .21133) 
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FIGURE  B-2:  ACTUAL  AND  ESTIMATED  TIME  TO  TAKE  LESSON  (TESTING  INCLUDED) 
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23  cases  plotted.  Regression  statistics  : 

Correlation  .04325  R  Squared  .00187  S.E.  of  Est  40.81798  Sig.  .8446 
Intercept (S.E.)  88.17899(  21.45858)  Slope(S.E.)  .05497(  .27710) 
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FIGURE  B-3.  ACTUAL  AND  ESTIMATED  TIME  TO  TAKE  LESSON  (TESTING  NOT  INCLUDED) 
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19  cases  plotted.  Regression  statistics: 
Correlation  .52781  R  Squared  .27858  S.E.  of  Est 
Intercept (S.E. )  34.17747(  7.28319)  Slope(S.E.) 


13.16558 

.23379 


Sig.  .0202 
(  .09125) 
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FIGURE  B-4.  ACTUAL  AND  ESTIMATED  TIME  TO  TAKE  LESSON  (TESTING  INCLUDED) 
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19  cases  plotted.  Regression  statistics: 
Correlation  .26276  R  Squared  .06905  S.E.  of  Est 
Intercept (S.E.)  64.27873  (  12.14134)  Slope(S.E.) 


21.94750  Sig.  .2771 
.17080(  .15211) 
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APPENDIX  C 


Mastery  Levels  Attained  by  Non-Masters  at  Pretest 

This  appendix  gives  training  managers  additional  information  about  the  attainment  of 
students  who  take  the  CBI  lessons  for  personnel  in  the  FSB.  In  the  body  of  this  report,  the 
criterion  for  mastery  was  set  at  80  percent  correct.  The  training  manager  may  wish  to  use  a 
different  criterion.  This  appendix  examines  the  attainment  of  students  who  scored  less  than 
70  percent  correct  on  the  pretest.  About  70  percent  of  the  students  who  participated  in  the 
beta  testing  of  these  lessons  attained  scores  in  this  range  on  the  pretest.  This  level  of 
performance  is  presumed  to  indicate  that  additional  training  in  these  areas  is  required.  This 
appendix  shows  what  percentage  of  these  students  (in  need  of  instruction)  attained  each  of 
several  levels  of  mastery. 

Table  C-1  displays  information  about  attainment  for  those  students  who  attained  less 
than  70  percent  correct  on  the  pretest  for  each  lesson.  For  each  of  six  levels  of  mastery 
attainment  (70%,  75%,  80%,  85%,  90%,  and  95%  correct  on  the  posttest),  the  percentage  of 
the  pretest  non-masters  who  attained  this  level  is  indicated  in  the  body  of  the  table.  Thus, 
100  percent  of  the  students  who  were  non-masters  at  pretest  on  Defend  Company  Area 
lessons  attained  a  score  of  75  percent  correct  on  the  posttest.  Only  27  percent  of  these 
students  attained  a  score  of  95  percent  correct  on  the  posttest.  Figure  C-1  displays  this 
information  graphically. 


Table  C-1:  Percentage  of  Students  in  Each  Lane  Attaining  Mastery,  by  Mastery  Cut  Score 

Mastery  Level  Cut  Score 
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43 

21 
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0 
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Percent  Mastery 


Lanes  Training  CBI 
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Figxjre  C-1.  Mastery  level  attainment  for  CBI  lessons  in  eaeh 
training  lane,  by  mastery  cut  score. 
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